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Distributed Item Review: Administrator User Guide 

The Distributed Item Review (DIR; Behavioral Research and Teaching, 2013) is a secure 
and flexible, web-based system designed to present test items to expert reviewers across a broad 
geographic area for evaluation of important dimensions of quality (e.g., alignment with 
standards, bias, sensitivity, and student accessibility). The DIR is comprised of essential features 
that allow system administrators to design and conduct customized review studies, in which test 
items are bulk uploaded, paired with target academic content/performance standards and images, 
and then organized and assigned for review by selected experts (e.g., content and field experts, 
education leaders and practitioners, education researchers, and test and curricula developers). 
Administrators can choose from predefined item review questions, or design new review 
questions pertinent to their study. A variety of administrator-designed support resources can be 
uploaded and associated with item review studies. Web-based reporting features offer 
administrators a “quick view” of reviewer progress and study completeness; comma separated 
value (.csv) reporting functionality allows data files to be securely downloaded and saved in a 
table-structured format for use in statistical analyses (e.g., SPSS, R). 

The following document provides detailed information about the DIR system, framed by 
a step-by-step administrator’s guide to successfully creating and conducting test item reviews 
using the DIR. A DIR administrator may alter the sequence of the steps outlined, and possibly 
omit or alter a given step, we present the steps in the order that follows to comprehensively and 
parsimoniously discuss the principle steps, having successfully conducted several review studies 
across diverse research contexts. This administrator user guide can be downloaded at: 
http://www.brtprojects.org/publications/technical-reports. 
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Prior to reading this user guide and attempting to create a DIR item review, we 
recommend prospective administrators view the DIR System Introduction — downloadable as a 
Portable Document Format (.pdf) at: http://www.brtprojects.org/publications/presentations. The 
slides give prospective DIR administrators an overview of the system, offering a jumpstart to 
proceeding through this guide. 

Step One: Registering a User Account in the DIR 

Prior to registering an account with the DIR, a prospective administrator or user should 
ensure their device meets the system requirements. The DIR system is largely platform 
independent (i.e., Windows, Android, Macintosh), functional across devices (e.g., desktop, 
laptop, tablet, and mobile), and compatible with common web browsers (e.g., Internet Explorer, 
Firefox, Chrome, Safari). Additionally, upon creating an account a DIR user assents to our User 
Agreement and Privacy Policy, which can be found along with a complete listing of system 
requirements at : http : //brtitemreview. com/ about#sy stem-requirements . 

To register a new user account with the DIR, click the “Register” button at the bottom of 
the DIR homepage (http://brtitemreview.com), displayed in the screenshot, below. On the New 
User Registration 
page, complete the 
required fields and 
click “Register”. 

Additional fields 
(e.g., Educator 
Role, Years 
Experience, 
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Degrees ) are helpful for disaggregating item response data, and should be required of all review 
study participants (see screenshot, below). 

Once a new 
user account has been 
registered, users 
receive an email 
welcoming them to the 
DIR system. If 
intending to register as 
an administrator, a 
researcher from 
Behavioral Research 
and Teaching (BRT) at 
the University of 

Oregon (http://www.brtprojects.org), the developers and managing directors of the DIR system, 
will adjust the status from user to administrator once approved, giving the ability to create and 
conduct item reviews in the DIR. Once a DIR administrator, reviewer user accounts are 
accessible through the “Users” button located in the left essential features panel. 

Step Two : Creating a Review in the DIR 

The DIR is comprised of several essential features. Reviews are the broadest essential 
feature, and can be thought of as the main house that frames other system features necessary for 
conducting item reviews using the DIR. Administrators create and define Reviews based on their 


New User Registration 

Thank you for offering to help improve academic assessments by providing your expert knowledge and feedback as an 

educator. For the best possible experience, please ensure you meet our system requirements. By creating an account, you 

agree to the terms and conditions of our Privacy Policy and User Agreement. 

First Name * 

[" First Name 

Last Name * 

Last Name 

Email Address * 

Email Address 

Password * 

Password 

Confirm Password * 

Password 

Educator Role 

General Education t 

School District 

School District 

Position 

Position 

Years Experience 

Years 

Degrees 

Degrees 

Address 

Address 

City 

City 

State 

State 

Zip Code 

Zip Code 





item review study’s design and goals. Once logged into the DIR, a panel of essential feature 
buttons always appears on the left side of the DIR Administration page (see screenshot, below). 
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HI Assignments 


Items 

1 o 

Questions 

<# Resources 

P 

Reviews 

0 

Standards 

.h 

Reports 

X 

Users 

0 

My Account 


Distributed Item Review Administration 


Listed below are the entities you can 


to manage your your item review. 


Assignments — Assign a subset of review items to a list of users for review 

Items — Add and edit items individually 

Questions — Manage the list of available review questions 

Resources — Add or edit resources to include in reviews 

Reviews — Manage item reviews, including start date, end date and available questions 
Standards — Create and edit item standards 
Reports — View and export reports of item reviews 
Users — Manage user accounts 


RBRT 


Copyright © 2014 - University of Oregon, Behavioral Research and Teaching 
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To create a new Review, click the “Reviews” button. A list of existing Reviews 
appears — click “Create Review” to navigate to the Reviews / Create page. Several fields define 
a Review. These fields are displayed in Table 1, along with a description of their purpose and 
functionality. All fields are typically completed to describe the item review study for reviewers. 
Table 1 


Review Field Names and Associated Descriptions 


Reviews / Create Field 

Description 

Subject 

Content area subject (English Language Arts, Math, or Science) 

Title 

Title of the item review (may house multiple Assignments — see 

Step 7) 

Date 

General date description for item review (e.g., Fall 2016) 

Start/End Dates 

Calendar dates that control when the item review study opens and 
closes to reviewers 

Grade(s) 

Description of the grade(s) level 

Resources 

List of support Resources — see Step 6 — that can be selected for 
use across Reviews and Assignments 

Review Instructions 

Instructions for the item review 

Email Notice Message 

Email notification that reviewers receive once they create a DIR 
user account 

Questions 

List of item review Questions — see Step 5 — that can be selected 
(drag/drop) for use across Reviews and Assignments 

Review Questions 

Item review Questions selected for a particular review 
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The Reviews / Create (or Edit ) page is shown, below. When populating fields that 
require substantial amounts of text (e.g., Review Instructions, Email Notice Message) 
administrators should draft the text in a word processing program, and then copy/paste the 
desired text into the target field, to better ensure accuracy. A screenshot of a sample Review with 
populated fields is shown, below. Note that the Review fields are populated through a 
combination of drop-down 
selection (e.g., Subject), point- 
and-click (e.g., Start / End Date, 

Resources), and typed-text (e.g., 

Title, Review Instructions) 
procedures. Note, also, that some 
Review fields require essential 
features be previously defined. 

For example, to select and order 
Review Questions (drag-and-drop 
from Questions), they must first 
be created in the DIR (see Step 5). 

Similarly, in order to select 
Resources, they must first be 
created and uploaded into the DIR (see Step 6). In other words, the DIR administrator creates a 
new Review in order to provide the framework for the study they are conducting, and because a 
Review must exist in order to create Items in the DIR (see Step 4); however, an administrator 
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typically returns to the Review / Edit page several times over the course of creating, refining, and 
finalizing a test item review study in the DIR. 

Step 3 : Creating and Uploading Standards in the DIR 

Typically, the essential feature addressed after creating a Review is creating/uploading 
academic content or performance Standards — created either individually or in bulk 
(recommended). To create Standards using the DIR’s bulk upload functionality, click the 
“Standards” button, located in the left essential features panel, to navigate to the Standards / Bulk 
Upload page (see screenshot, below). The administrator must create a .csv spreadsheet (for 
example, using Excel) with four column headers: Standard ID, Subject, Domain, and 
Description. The Standard ID and Subject fields are required to contain information in the .csv 



spreadsheet. Domain and Description, while not required to contain information, are almost 
always populated in the .csv, as they typically convey relevant information for reviewers 
evaluating the alignment between test items and target standards. A portion of a sample .csv is 
shown in the screenshot, below. Note, in this example the optional Domain and Description 
fields contain information. The two Math standards shown are from the Oregon Extended 
Assessment, for which we developed essentialized standards by decreasing relevant Common 
Core State Standards (CCSS) in depth, breath, and complexity. Therefore, in this sample .csv 
information in the Domain field represents the CCSSs standard, and information in the 
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Description field 
represents the 
essentialized standard at 
three levels of 
complexity (Low [L], 
Medium [M], and High 



A 

B 

c 

D 

1 E | 

F 1 

1 

Standard ID 

Subject 

Domain 

Description 



■ 1 

22 

M03GEO1.1 

math 

Use attributes of triangles, 
squares, and circles to classify 
shapes. 

L - identify triangles (all shapes in 
answer choices same-size); M - 
identify squares (shapes in answer 
choices of various sizes); H - 
identify circles (shapes in answer 
choices of various sizes). 
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M03GEO1.2 

math 

Use unit squares to determine 

1/2 or the whole. 

L- use unit squares to identify 
whole areas shaded up to 2X2; M - 
use unit squares to identify whole 
or 1/2 areas shaded up to 3X3 
(with shading done only one side); 
H - use unit squares to identify 
whole areas shaded up to 4X4 or 
1/2 of any square figure up to 4X4 
(with shading done on diagonals). 




[H]). In a different review study, the Domain field might represent the CCSS domain (e.g., 
Geometry, Functions), while the Description field would then represent the actual CCSS 
standard. Thus, the type of Standards created for the test item review, and therefore, the way in 
which the .csv spreadsheet is populated with information, depends on the particular review study 
that the administrator is conducting in the DIR. 

Each row in the .csv spreadsheet represents a single academic content or performance 
standard. Of critical importance is that each Standard ID is unique, so that it can be 
associated with a target Item. Typically, Item IDs contain the target Standard ID to which it is 
aligned. Item naming convention is discussed in greater detail, below (see Step 4). Once the 
.csv is populated with all standards, including across Subjects (currently, english, math, or 
science ) if necessary, the .csv file is uploaded by clicking on the “Choose File” button, 
navigating to the 


location of the 
.csv, selecting it, 
and then clicking 
“Upload” on the 
Standards / Bulk 


Standards 


j Create Standard 


Bulk Upload 



Subject Standard ID Domain Standards 

Math M03GEO1 1 Use attributes of trian 9 |es > squares, and L - identify triangles (all shapes in answer choices same-size); M - identify squares (shapes in answer 

circles to classify shapes. choices of various sizes); H - identify circles (shapes in answer choices of various sizes). 

L- use unit squares to identify whole areas shaded up to 2X2; M - use unit squares to identify whole 
Math M03GEO1 2 Use unit s 9 uares t0 determine 1/2 or the or 1/2 areas shaded up to 3X3 (with shading done only one side); H - use unit squares to identify 

whole. whole areas shaded up to 4X4 or 1/2 of any square figure up to 4X4 (with shading done on 

diagonals). 


Edit Delete 


Math M04GEO1 .1 Identify points, line segments, and angles. 


Math M04GEO1.2 


Identify triangles, circles, squares, and 


L - Given a point, line, and angle, student identifies point; M - identify line segments; H - identify 


L - identify triangles; M - identify squares and circles; H - identify rectangles. 
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Upload page. Once uploaded into the DIR, an administrator should consider spot-checking 
several standards to ensure correctness. Simply, click on the “Standards” button located in the 
left essential features panel to navigate to the Standards page, and then click on a particular 
standard (see screenshot, above). Completing spot-checks is particularly important for standards 
that contain non-alphabetic symbols (e.g., division/square root symbols in math standards) as 
they may display improperly during bulk upload from the .csv to the DIR. 

Step 4 : Creating and uploading Items in the DIR 

Once Standards are uploaded into the DIR, the administrator typically creates Items, and 
associates them with target standards. Like Standards, Items can be created either individually 
or in bulk (recommended). To create Items using the DIR’s bulk upload functionality, click the 
“Items” button located in the left essential features panel, to navigate to the Items / Bulk Upload 
page (see screenshot, below). 

Items / Bulk Upload 

To upload items in bulk, the Items File must be in .csv format and contain the following columns in order (optional fields may be left blank): 

• Item ID - Unique item identifier 

• Group - (optional) Item group / task 

• Standards - A colon-delimited set of standards 

• Item Information - (optional) Scoring protocol and information 

• Prompt - (optional) Item prompt text 

• Option A - (optional) Response option A text 

• Option B - (optional) Response option B text 

• Option C - (optional) Response option C text 

• Correct - (optional) Letter of correct response option (a, b, or c); 'a' is assumed to be correct if missing 

To include images for items, the Images Archive mush be an archive in .zip format containing the images. Item images must have a file name of the corresponding Item ID, while item option images 

must have a file name of the corresponding Item ID followed by an underscore followed by the option letter. Accepted file formats are JPEG, PNG, GIF, and SVG. 

Example: For an item with ID '123XYZ', the following are valid image file names: 123XYZ.jpg, 123XYZ_a.png, 123XYZ_b.svg, and 123XYZ_c.gif. 

Review 

ORExt English Language Arts Items ± 

Items File (.csv) 

choose File no file selected 

Images Archive (.zip) 

choose File no file selected 


Upload 


The administrator must create a .csv spreadsheet with nine column headers (fields): Item 
ID, Group, Standards, Item Information, Prompt, Option A, Option B, Option C, and Correct. 
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Column fields a description of their purpose and functionality are displayed in Table 2 — also 
shown on the Items / Bulk Upload page in the DIR. Item ID and Standards fields are required to 
contain information in the .csv; populating additional column fields once again depends on the 
nature of the test items to be reviewed in the DIR. 

Table 2 


Items .csv Field Names and Associated Descriptions 


Items / Bulk Upload 

Description 

Item ID (required) 

Unique item identifier 

Group (optional) 

Item group / task (e.g., Grade level of test item) 

Standards (required) 

A colon-delimited standards identifier that ideally links to Item ID 

Item Information (optional) 

Scoring protocol and information (i.e., for test administrator 
booklets) 

Prompt (optional) 

Item prompt text (i.e., test item prompt that the student sees in 
their testing booklet) 

Option A (optional) 

Response option A text (i.e., for selected-response item types) 

Option B (optional) 

Response option B text (i.e., for selected-response item types) 

Option C (optional) 

Response option C text (i.e., for selected-response item types) 

Correct (optional) 

Letter of correct response option (a, b, or c); 'a' is assumed to be 
correct if field is missing information 


For example, if test items exist in the form of image files (e.g., screenshots of items from 
a computer-based testing environment, or screenshots of items from a .pdf /.docx of a testing 
booklet), the DIR administrator could follow these seven steps — essentially providing 
information in the Item ID, Standards, and possibly the Correct column fields in the .csv — to 
upload and associate Items with corresponding Standards and images: 

1. Populate the Item ID and Standards, fields for all items (rows) in the .csv. 
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2. Create a zipped folder (.zip) containing the image files (e.g., .jpg, .png, .gif, .svg) with 
individual file names of the corresponding Item ID. 

3. Optionally, for items with three response options (i.e., like the Oregon Extended 
Assessment) the administrator can populate the Correct field with the correct response 
option for each test item (a, b, or c). 


4. Select the proper Review from the 
drop-down menu on the Items / Bulk 
Upload page. 

5. Upload the .csv file by clicking on the 
“Choose File” button under Items 

File, navigating to the location of the .csv, and selecting it. 

6. Upload the .zip folder 
containing the test item 
images by clicking on the 
“Choose File” button 


Item 4 

Which is an animal? 



! 


A 







horse 

mushroom 

tree 



under Images Archive, 
navigating to the .csv 
location, and selecting it. 

7. Click “Upload” at the 
bottom of the Items / Bulk 
Upload page. 

A sample screenshot of an 
image file of a publically released 
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Grade 5 practice science item from the Oregon Extended Assessment is shown above, followed 
by a screenshot just below of what the item looks like from a reviewer’s perspective once it is 
uploaded into the DIR and associated with a standard and image file following the seven steps 
enumerated above. Note that additional DIR essential features are also shown in the latter 
screenshot (i.e., Questions and Resources), which are discussed in greater detail, below (see 
Steps 5 and 6, respectively). 

Alternatively, if test items do not yet exist in an electronic form like an image file, an 
administrator could populate the additional column fields in the Items .csv, and essentially build 
the test items in the DIR. In the screenshot shown below, all column fields in a sample .csv are 


_1 

Item ID 

1 B 

c 

D 

E 

F 

G 

H 

1 

1 

Group 

Standards 

Item Information 

Prompt 

Option A 

Option B 

Option C 

Correct 

2 

M030AT1.1_L04 

3 

M03OAT1.1 

(L) Here is a feather on a hat. This 
hat has one feather. (Point to the 
student materials.) If you have 
one hat, how many feathers do 
you have: 1, 2, or 3? 

If you have one hat, how 
many feathers do you have? 

1 

2 

3 

a 

3 

M030AT1.2_L04 

3 

M03OAT1.2 

(L) Here are 4 hats with a dashed 
line between them. (Point to 
student materials.) How many 
hats are on one side on the 
dashed line: 1, 2, or 3? 

How many hats are on one 
side on the dashed line? 

1 

2 

3 

b 

4 

M030AT4.9_L02 

3 

M030AT4.9 

(L) Here are some grapes. (Point 
to student materials.) How many 
grapes are there: 5, 6, or 7? 

How many grapes are there? 

5 

6 

7 

c 

5 

M03NBT1.2_L03 

3 

M03NBT1.2 

(L) Here is an addition problem. 
(Point to student materials.) 6 + 8 
= which number: 8, 10, or 14? 

6 + 8= ? 

8 

10 

14 

c 

6 

M030AT1.2_M08 

3 

M030AT1.2 

(M) Here are two trees with some 
apples. (Point to student 
materials.) There are 8 apples all 
together. How many apples are 
on one tree: 4, 5, or 6? 

How many apples are on 
one tree? 

4 

5 

6 

a 


populated. This particular .csv was used to build and bulk upload publically released Grade 3 
practice math items for the Oregon Extended Assessment. Notice these math items are selected- 
response item types, with three answer options (a, b, and c). Thus, it is worth emphasizing once 
again that the manner in which an administrator uses the .csv to create Items in the DIR is wholly 
dependent on the nature of the test items to be reviewed in the DIR. 

Additionally, in the sample .csv above, note the similarity between the information in the 
Item ID and paired Standard fields. Earlier, it was mentioned that the Item ID typically contains 
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the associated Standard ID to which it is aligned, and that is the case in this example. The first 
test item in the above .csv (row 2) is identified using an Item ID of M030AT1.1L04. The first 
9 characters of that Item ID are the same (in order) as the associated Standard to which it is 
aligned, where: M= Math, 03 = Grade 3, OAT1.1 = Operations and Algebraic Thinking 
(essentialized CCSS domain/standard), and L04 = Low complexity level item #4 (a coding 
particular to the Oregon Extended Assessment). Thus, for the purpose of conducting test item 
reviews in the DIR it is critical that both Item IDs and Standard IDs are unique. Further, it 
is highly recommended that these unique identifiers consist of relevant information that 
conspicuously associates test Items with their target Standards. In the example Item ID, the 
test item identifier (M030AT1.1L04) provides relevant information such as content area, grade, 
and complexity levels, specifically contains the standard identifier, and yet, is unique from other 
items in a manner that maintains database integrity for later analysis of test item review data. 

Step 5 : Creating Questions in the DIR 

Review Questions are study-specific, and once created by administrators, are selected and 
assigned to all Items in Reviews from the Review / Create or Edit pages (see Step 2, above). 
Questions are customizable, and allow test items to be reviewed by experts for important 
dimensions of quality, including alignment with standards, bias, sensitivity, and student 
accessibility. Current response option formats include: 

• Yes/No (binary) e.g., Item is free of bias, 

• Numeric (Likert / ordinal rating scale) e.g., Rate the strength of alignment between the 
test item and standard/s, 

• Text Box (single-line constructed response) e.g., Reword the item prompt to be more 
inclusive of diverse populations, and 
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• Comment (multiple-line constructed response) e.g., For any "0" or "No" rating, please 
provide a rationale and recommendation/s for improving the item. 

To create Questions, click the “Questions” button, located in the left essential features 
panel, to navigate to the Questions page (see screenshot, below). Next, click the “Create 


Questions 



5 Records 




Type 

Text 

Format 

Edit Delete 

Standard 

Rate the strength of alignment between the test item and standard/s. 

Numeric 

/ 3 

Item 

Item is Free of Bias 

Boolean 

/ □ 

Item 

Item is Accessible to SPED Students 

Boolean 

/ ^ 

Item 

Comments 

Comment 


Item 

For any "0“ or "No" rating, please provide a rationale and recommendation/s for improving the item. 

Comment 

/ 




Question” button to navigate to the Questions / Create page. Select Type and the response 
Format, add relevant Text, including any text explicative of numeric response options, and then 
click “Create” to save the question. An example Question with a three-point numeric response 
option is shown in the screenshot, below. Once created, Questions are selected from the Review 
/ Create or Edit page using the DIR’s drag-and-drop functionality (see Step 2). 


Questions / Create 


Type 

Standard 


Rate the strength of alignment between the test item and standard/s.| 


Format 

Numeric 


# Description 


Not Aligned 


Somewhat Aligned 
Aligned 


Create 
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Step 6 : Creating Resources in the DIR 

Resources are study-specific, and once created by administrators, are selected from the 
Review / Create or Edit pages (see Step 2). The purpose of the Resources essential feature is to 
allow DIR administrators to create/upload electronic resources that are pertinent to their research 
study and that support reviewers in the process of examining test item quality. Current 
Resources formats include, but are not necessarily limited to: 

• PDFs (e.g., reviewer/study training slides, academic content/performance standards, state 
accessibility/accommodations manuals), 

• Word (or other word processing) documents (e.g., item-development templates), 

• Videos (e.g., reviewer/study training webinars, representative testing population, test 
administration examples), and 

• Image files (e.g., standard essentialization flow chart, sample item images). 

To create Resources, click the “Resources” button, located in the left essential features 
panel, to navigate to the Resources page (see screenshot, below). Next, click the “Create 
Resource” button to navigate to the Resources / Create page. Enter a Title for the Resource, and 
add any relevant descriptive text to explain what the Resource is and how it should be used 


Resources 9 Records 





Title 

Created By 

Edit Delete 

0 Video of Representative Student Population 

Steve J. 

/ — j 

li ORExt Item Development and Specifications 201 4-201 5 

P. Shawn 1. 

/ 1 □ 

t Oregon Accessibility Manual: 2014-2015 School Year 

P. Shawn 1. 

/ -j 

li The First Contact Census Student Characteristics: Supplemental Handout 

P. Shawn 1. 

/ 

li DIR Webinar Training Slides 

P. Shawn 1. 

/ u 

li ORExt Essentialization Flowchart 

P. Shawn 1. 

/ □ 1 

0 DIR Training Webinar Recording 

Steve J. 

/ ~\ 

Q PASA Essentialization Process 

Dan F. 

/ □ 

0 PASA Item Writer Training PPT 

Dan F. 

/ ~) 
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in the context of reviewing test items. Click on the “Choose File” button, navigate to the 
location of the desired Resource, and then click “Create” to upload and save the resource in the 
DIR. An example video Resource is shown in the screenshot, below. Once created, Resources 
are selected from the Review / Create or Edit by clicking on the bubble next to each Resource 
pertinent to a particular Review study (see Step 2). 

Resources / Edit 

Title 

Video of Representative Student Population 

Description 

Video of seven different students who represent the student population that participate in Oregon's Extended Assessment. Please note, this video must be viewed using Safari, Internet 
Explorer or Google Chrome - Firefox will not work. 

,g| 

File (OregonExtended_StudentPopulationsV2small.m4v) 
choose File no file selected 


Update 


Step 7 : Creating Review Assignments in the DIR 

Once the preceding essential features relevant to a particular review study are 
created/uploaded into the DIR (i.e., Reviews, Items, Standards, Questions, and Resources), and 
once all prospective reviewers have registered DIR User accounts, Assignments are created. 
Assignments directly reflect the review study design, including how study participants 
(reviewers) are matched to test items. Essentially Assignments allow DIR administrators to: 

• Bulk select/group and order test Items (and therefore, associated standards/images) by 
content area, grade, and/or other item-level characteristics, 

• Select and designate reviewers to selected/grouped Items, and 

• Provide assignment-specific directions to reviewers. 

To create an Assignment, click the “Assignments” button, located in the left essential 
features panel, to navigate to the Assignments page (see screenshot, below). Next, click the 
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“Create Assignment” button to navigate to the Assignments / Create or Edit page. Using the 
drop-down menu, select the Review that will house the Assignment to be created, enter a Title, 
and add any relevant descriptive text to explain what the Assignment is, and how it should be 
completed, including any specific recommendations or suggestions necessary for completing the 
Assignment review successfully. In the Reviewers box, click the boxes next to the names of each 
reviewer assigned to the group of Items associated with the Assignment. 


Assignments 1 9 Records 


Create Assignment Delete 


Review 

Title 

Closing Date 

A 

Edit 

Delete 

ORExt English Language Arts Items 

All ELA Items 

03/05/2016 

5 

/ 


ORExt Science Items 

Grade 5 Science Item Review 

03/05/2016 

4 

/ 

□ 

ORExt Science Items 

Grade 8 Science Item Review 

03/05/2016 

3 

/ 

□ 

ORExt Science Items 

Grade 1 1 Science Item Review 

03/05/2016 

3 

/ 

□ 

ORExt Math Items 

Grade 3 Math Item Review 

03/26/2016 

3 

/ 

n 

ORExt Math Items 

Grade 4 Math Item Review 

03/26/2016 

3 

/ 

□ 

ORExt Math Items 

Grade 5 Math Item Review 

03/26/2016 

4 

/ 

□ 

ORExt Math Items 

Grade 6 Math Item Review 

03/26/2016 

3 

/ 

D 

ORExt Math Items 

Grade 7 Math Item Review 

03/26/2016 

3 

/ 

□ 

ORExt Math Items 

Grade 8 Math Item Review 

03/26/2016 

3 

/ 

n 

ORExt Math Items 

Grade 1 1 Math Item Review 

03/26/2016 

3 

/ 

□ 


To select and order Items to appear in a given Assignment (and Review ), use the drag- 
and-drop functionality to move test items from the Items box to the Assignment Items box in the 
exact order desired. The Paste Item IDs functionality allows Item IDs to be organized in bulk in 
the desired order in another program (e.g., Excel), and then copy/pasted into the Assignment 
Items box — convenient for Assignments consisting of many Items. An example Assignment, with 
reviewers selected (Steve and Shawn, a special educator and a general educator, respectively), is 
shown in the screenshot, below. Based on the Review and Title fields this Assignment is 
organized by grade (Grade 5) and content area (science). Additionally, Items were selected using 
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the bulk ordering feature through an Excel spreadsheet consisting of the relevant Item IDs — this 
particular Assignment consisted of 496 Items, with a “sister” Assignment consisting of the same 
Items, counter-balanced (reverse-ordered) in their presentation to reviewers. 

Assignments / Edit 

Review 

ORExt Science Items J 

Title 

Grade 5 Science Item Review 

Description 

Included in this assignment are 495 individual science items. Please review all items making sure to take breaks as necessary, so that you review each item carefully and completely. 

Please write all comments and suggestions in the Comments window below each set of questions. Your comments are critical to help us make an item aligned to an essentialized standard 
when you rate an item alignment as a 0. Also, if you have any suggestions for making the item more accessible to students with significant cognitive disabilities, please include those in the ^ 

Reviewers 

Q Jonas, Steve <sjinor@yahoo.com> 

Glasgow, Aaron <aglasgow@uoregon.edu> 

H Cords, Trevor <tcords@gmail.com> 

H Megert, Raina <rainam@uoregon.edu> 

3 Irvin, Shawn <p.shawn.irvin@gmail.com> 

T Aazzerah, Rachel <rachel.aazzerah@state.or.us> 


Items Assignment Items (in order) 


S08PHS1 .2LSAMPLE 


1. 

S05PHS1 .1 LSAMPLE 

S08PHS1.2L01 


2. 

S05PHS1.1L01 






Step 8 : Using DIR Reports Features 

The DIR offers two types of Reports features: (a) an HTML (web-based) report that 
offers administrators a “quick view” of reviewer progress, and thus, Review! Assignment 
completeness, and (b) a .csv report that allows data files to be securely downloaded and saved in 
a table- structured format 
for import into spreadsheet 
and statistical programs 
(e.g., Excel, SPSS, R). 

Access both types of 
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Reports by clicking on the “Reports” button located in the left essential features panel, and 
navigating to the Reports page (see screenshot, above). Reports are organized by Review, and 
then by Assignment (once a Review is selected). An example of an HTML, or “quick view”, 
report is shown in the screenshot, below, followed by a sample .csv report. 


ORExt English Language Arts Items 


02 / 27 / 2016 - 03 / 05/2016 


All ELA Items 

Jonas, Steve 

0% 

a 

Cords, Trevor 

0% 


Megert, Raina 



Farley, Dan 

0% 


Tindal, Gerald 

0% 





1 B 

c 

D 

E 

F G H 1 j K L 

i 

First Name 

Last Name 

Email 

Assignment 

Item ID 

Question Response 

2 

Reviewer 1 

Reviewer 1 

Revl@egdistrictl.com 

Grade 11 SciltemRev 

S11ESS1.1H10 

Rate the strengt 1 - Sufficient alignment 

3 

Reviewer 1 

Reviewer 1 

Revl@egdistrictl.com 

Grade 11 SciltemRev 

S11ESS1.1H10 

Item is Free of B Yes 

4 

Reviewer 1 

Reviewer 1 

Revl@egdistrictl.com 

Grade 11 SciltemRev 

S11ESS1.1H10 

Item is Accessibl Yes 

5 

Reviewer 1 

Reviewer 1 

Revl@egdistrictl.com 

Grade 11 SciltemRev 

S11ESS1.1H10 

For any "0" or "f Image is a bit too busy - consider getting rid of some surronding stars. 

6 

Reviewer 1 

Reviewer 1 

Revl@egdistrictl.com 

Grade 11 SciltemRev 

S11ESS1.1H14 

Rate the strengt 2 - Strong alignment 

7 

Reviewer 1 

Reviewer 1 

Revl@egdistrictl.com 

Grade 11 SciltemRev 

S11ESS1.1H14 

Item is Free of B Yes 

8 

Reviewer 1 

Reviewer 1 

Revl@egdistrictl.com 

Grade 11 SciltemRev 

S11ESS1.1H14 

Item is Accessibl Yes 

9 

Reviewer 1 

Reviewer 1 

Revl@egdistrictl.com 

Grade 11 SciltemRev 

S11ESS1.1H14 

For any "0" or "No" rating, please provide a rationale and recommendation/s for improving the item. 

10 

Reviewer 2 

Reviewer 2 

Rev2@egdistrict2.com 

Grade 11 SciltemRev 

S11ESS1.1H10 

Rate the strengt 2 - Strong alignment 

11 

Reviewer 2 

Reviewer 2 

Rev2@egdistrict2.com 

Grade 11 SciltemRev 

S11ESS1.1H10 

Item is Free of B Yes 

12 

Reviewer 2 

Reviewer 2 

Rev2@egdistrict2.com 

Grade 11 SciltemRev 

S11ESS1.1H10 

Item is Accessibl Yes 

13 

Reviewer 2 

Reviewer 2 

Rev2@egdistrict2.com 

Grade 11 SciltemRev 

S11ESS1.1H10 

For any "0" or "No" rating, please provide a rationale and recommendation/s for improving the item. 

14 

Reviewer 2 

Reviewer 2 

Rev2@egdistrict2.com 

Grade 11 SciltemRev 

S11ESS1.1H14 

Rate the strengt 2 - Strong alignment 

15 

Reviewer 2 

Reviewer 2 

Rev2@egdistrict2.com 

Grade 11 SciltemRev 

S11ESS1.1H14 

Item is Free of B Yes 

16 

Reviewer 2 

Reviewer 2 

Rev2@egdistrict2.com 

Grade 11 SciltemRev 

S11ESS1.1H14 

Item is Accessibl Yes 

17 

Reviewer 2 

Reviewer 2 

Rev2@egdistrict2.com 

Grade 11 SciltemRev 

S11ESS1.1H14 

For any "0" or "No" rating, please provide a rationale and recommendation/s for improving the item. 

18 

Reviewer 3 

Reviewer 3 

Rev3@egdistrict3.com 

Grade 11 SciltemRev 

S11ESS1.1H10 

Rate the strengt 2 - Strong alignment 

19 

Reviewer 3 

Reviewer 3 

Rev3@egdistrict3.com 

Grade 11 SciltemRev 

S11ESS1.1H10 

Item is Free of B Yes 

20 

Reviewer 3 

Reviewer 3 

Rev3@egdistrict3.com 

Grade 11 SciltemRev 

S11ESS1.1H10 

Item is Accessibl Yes 

21 

Reviewer 3 

Reviewer 3 

Rev3@egdistrict3.com 

Grade 11 SciltemRev 

S11ESS1.1H10 

For any "0" or "No" rating, please provide a rationale and recommendation/s for improving the item. 

22 

Reviewer 3 

Reviewer 3 

Rev3@egdistrict3.com 

Grade 11 SciltemRev 

S11ESS1.1H14 

Rate the strengt 2 - Strong alignment 

23 

Reviewer 3 

Reviewer 3 

Rev3@egdistrict3.com 

Grade 11 SciltemRev 

S11ESS1.1H14 

Item is Free of B Yes 

24 

Reviewer 3 

Reviewer 3 

Rev3@egdistrict3.com 

Grade 11 SciltemRev 

S11ESS1.1H14 

Item is Accessibl Yes 

25 

Reviewer 3 

Reviewer 3 

Rev3@egdistrict3.com 

Grade 11 SciltemRev 

S11ESS1.1H14 

For any "0" or "No" rating, please provide a rationale and recommendation/s for improving the item. 

26 

Reviewer 4 

Reviewer 4 

Rev4@egdistrict4.com 

Grade 11 SciltemRev 

S11ESS1.1H10 

Rate the strengt 1 - Sufficient alignment 

27 

Reviewer 4 

Reviewer 4 

Rev4@egdistrict4.com 

Grade 11 SciltemRev 

S11ESS1.1H10 

Item is Free of B Yes 

28 

Reviewer 4 

Reviewer 4 

Rev4@egdistrict4.com 

Grade 11 SciltemRev 

S11ESS1.1H10 

Item is Accessibl Yes 

29 

Reviewer 4 

Reviewer 4 

Rev4@egdistrict4.com 

Grade 11 SciltemRev 

S11ESS1.1H10 

For any "0" or "f Increase size of the close-up of fusion. 

30 

Reviewer 4 

Reviewer 4 

Rev4@egdistrict4.com 

Grade 11 SciltemRev 

S11ESS1.1H14 

Rate the strengt 2 - Strong alignment 

31 

Reviewer 4 

Reviewer 4 

Rev4@egdistrict4.com 

Grade 11 SciltemRev 

S11ESS1.1H14 

Item is Free of B Yes 

32 

Reviewer 4 

Reviewer 4 

Rev4@egdistrict4.com 

Grade 11 SciltemRev 

S11ESS1.1H14 

Item is Accessibl Yes 

33 

Reviewer 4 

Reviewer 4 

Rev4@egdistrict4.com 

Grade 11 SciltemRev 

S11ESS1.1H14 

For any "0" or "No" rating, please provide a rationale and recommendation/s for improving the item. 


Summary and Contact Information 

The DIR is a flexible, online system designed to securely distribute test items to experts 
for review of important dimensions of quality (e.g., alignment with standards, bias, sensitivity, 
and student accessibility). The DIR’s essential features allow system administrators to design 
and conduct customized review studies, in which test items are bulk uploaded, associated with 
academic content/performance standards and item imagery, and then organized and assigned for 
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evaluation by recruited reviewers (e.g., content and field experts, education leaders and 
practitioners, education researchers, and test and curricula developers). A variety of resources 
can be uploaded and associated with item review studies to support reviewers in their 
examination of test items. Reporting features allow administrators to quickly assess reviewer 
progress and study completeness, or securely download review data in a table-structured format 
for use in statistical analyses (HTML and .csv, respectively). For more information about the 
DIR, or to discuss use of the system, please contact Dr. P. Shawn Irvin at BRT by email at 
pirvin@uoregon.edu, or phone at (541) 346-3535. 
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